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ABSTRACT 

A  growing  area  of  interest  in  the  sensor 
community  is  the  use  of  measurements 
made  at  multiple  wavebands  to  increase  the 
detection  and  acquisition  of  low  contrast 
targets.  Hyperspectral  imagery  involves 
large  amounts  of  high  resolution  spatial  and 
spectral  data  which  must  be  processed  in  a 
timely  manner  to  assist  the  decision  maker. 

A  lot  of  this  data  comes  in  the  form  of 
imagery  at  different  wavebands;  an  infrared 
camera,  synthetic  aperture  radar,  passive 
millimeter  wave  camera  or  some  other 
sensor.  This  paper  presents  a  new 
approach  for  combining  hyperspectral 
imagery  in  an  efficient  manner  to 
significantly  reduce  the  burden  on  the 
human  or  computer  decision  maker.  This 
technique  is  based  on  modern  image 
processing  algorithms  to  create  a  color 
image  where  color  is  directly  related  to 
waveband.  Multiple  two  dimensional  images 
are  combined  into  a  single  three  dimensional 
image  which  can  be  processed  in  a  single 
processor  instead  of  multiple  processors 
reducing  overall  sensor  cost  and  complexity. 
The  proposed  technique  has  application  to 
the  commercial  digital  camera  market  in 
addition  to  the  military  market. 


INTRODUCTION 

There  is  significant  interest  in  hyperspectral 
processing  in  the  infrared  (IR)  community  for 
background  suppression  and  noise 
reduction  to  increase  the  acquisition  range 
and  reduce  the  false  alarm  rate  of  infrared 
sensors.  This  work  is  in  the  relatively  early 
stages  of  development  with  efforts 
concentrating  on  understanding  the  basic 
phenomenon.  Applications  span  missile 
seekers,  threat  warning  systems  and 
acquisition  sensors. 


Hyperspectral  processing  holds  the  potential 
for  both  background  suppression  and  noise 
reduction.  Background  suppression  is 
achieved  because  the  target  signature 
changes  differently  from  one  spectral  band 
to  another  compared  to  background  and  this 
can  is  used  as  a  discriminant  for  attenuating 
background  energy.  Noise  is  suppressed  by 
exploiting  the  fact  that  target  is  correlated 
from  band  to  band  and  noise  is  uncorrelated. 
The  type  of  detection  processing  that  is 
performed  depends  on  whether  the  sensor  is 
noise  limited  or  background  limited  for  any 
particular  scene.  The  multiple  band  images 
also  reduces  the  likelihood  that  a  target  falls 
on  a  bad  pixel  and  is  not  seen  at  all. 

One  of  the  questions  being  asked  by 
researchers  working  in  the  area  of 
hyperspectral  processing  is  whether  the 
multiple  sensor  data  should  be  conbined  and 
then  processed  or  processed  and  then  the 
processed  results  combined.  This  alternate 
approach  to  fusing  the  hyperspectral  data  is 
illustrated  in  Figure  1 .  Figure  1  (a) 
illustrates  processing  followed  by  fusion 
while  Figure  1  (b)  illustrates  fusion  followed 
by  processing.  Which  approach  is 
is  best  for  any  application  depends  on  many 
system  requirements.  This  paper  discusses 
a  new  technique  for  combining  the  sensor 
data  prior  to  processing,  Figure  1(a). 

APPROACHES  TO  HYPERSPECTRAL 
DISCRIMINATION 

Hyperspectral  discrimination  exploits  a 
variation  in  one  of  the  radiation  parameters 
with  wavelength.  An  example  is  a  rocket 
plume  which  emit  differently  depending  on 
the  exhaust  gases  that  make  up  the  plume. 
Reference  1  list  some  of  the  emission  bands 
associated  with  typical  motor  exhaust  gases. 
For  example,  CO  emits  at  4.663p,  2.345p 
and  1 .573p  so  it  might  be  advantageous  to 
center  one  band 
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Figure  1 .  Input  Level  And  Output  Level  Sensor  Fusion 


about  4.663p  and  compare  it  to  one  at 
3.2p  where  CO  does  not  emit.  Both  these 
sub-bands  are  in  the  midwave  region  of  the 
IR  spectrum.  It  is  expected  that  the  amount 
of  energy  in  these  two  sub-bands  for  any 
particular  pixel  will  differ  depending  on 
whether  the  pixel  contains  a  rocket  plume 
exhausting  CO  plus  background  or 
background  only. 

Hyperspectral  discrimination  is  not  limited  to 
applications  where  the  target  contains 
exhaust  gases  of  know  constituents.  The 
reflectance  and  emissivity  of  materials 
chai  iges  with  wavelegth'.  If  it  changes 
differently  for  targets  materials  of  interest 
than  it  does  u  '  Aground  materials,  this 
serves  as  a  discriminant  to  separate  targets 
from  background.  This  phenomenon. is  less 
understood  and  is  a  driving  reason  for 
building  development  cameras  so  a 
database  can  be  assembled  and  used  to 
supports  discrimination  algorithm 
development. 

When  light  interacts  with  any  material  it  is 
absorbed,  reflected,  and  scattered. 
Absorbed  energy  is  ultimately  re-emitted  as 
thermal  energy  in  wavelength  region  that 
corresponds  to  the  temperature  of  the 
material  and  it’s  absorption  spectrum.  The 
infrared  emissivity  and  reflectivity  of 
materials  is  controlled  by  both  the  intrinsic 
optical  properties  of  the  material  itself  and 


the  morphology  of  the  material  surface. 
Typically  all  reflectivity,  absorption,  and 
emission  measurements  are  made  on  flat 
samples  of  the  material  with  polished  plane 
parallel  sides.  In  this  case  scatter  is 
hopefully  minimized  and  the  optical  budget 
for  the  material  can  be  determined.  For  low 
absorbing  materials,  emission  is  considered 
equal  to  absorption  and  generally  follows  the 
same  spectral  pattern  in  the  proper  thermal 
wavelength  band.  For  highly  absorbing 
materials  and  most  very  hot  materials, 
emission  follows  a  given  spectral  distribution 
defined  by  the  temperature  of  the  material 
which  is  described  as  the  black  body 
radiati  A'  curve.  Other  partially  absorbing 
materials,  or  even  low  absorbing  materials 
with  high  surface  roughness  can  have  an 
emission  spectrum  defined  by  a  combination 
of  the  material  properties  and  the 
temperature  and  these  are  generally 
described  by  the  "Gray  Body"  emission 
curve. 

Trying  to  identify  natural  versus  man  made 
materials  by  their  emission  spectrum  is  a 
highly  empirical  activity.  A  steel/  ceramic/ 
alloy  vehicle  in  the  woods  may  have  a 
painted  surface  that  provides  the  same 
emission  spectrum  as  wood  bark.  The  main 
difference  may  be  the  angular  distribution  of 
the  reflected  and  emitted  radiation  rather 
than  the  spectral  distribution,  assuming  that 
the  tree  and  the  vehicle  are  at  the  same 
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temperature.  If  the  vehicle  is  hot  in 
reference  to  it's  background  it  would  be 
easily  separated  out  from  the  background 
unless  the  surface  had  been  painted  with  a 
low  emissivity  material  to  compensate  for  the 
temperature  difference. 

Figure  2  shows  two  sets  of  curves  of 
reflectivity  versus  wavelength  taken  from 
Reference  2.  Figure  2(a)  was  selected  since 
it  shows  man  made  objects  and  Figure  2(b) 
was  selected  because  it  shows  natural 
objects.  Significant  changes  in  reflectivity 
can  be  seen  even  within  the  midwave 


Infrared  band.  Notice  that  some  of  the 
metals  in  Figure  2(a)  have  a  significant 
response  in  the  long  wave  band  while  the 
trees  in  Figure  2(b)  have  a  low  response  in 
the  longwave  region.  This  opens  up  the 
possibility  that  a  midwave/longwave  sensor 
may  be  able  to  detect  a  metal  object  near 
trees  more  easily  than  a  single  spectral 
sensor.  Further  study  is  required  to 
determine  the  utility  of  these  changes  for 
acquisition  and  the  best  way  to  exploit  any 
discriminant  derived  from  the  changes. 


(a) 


(b) 

Figure  2.  Change  in  Reflectivity  with  Wavelength  (Taken  from  Reference  2) 
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Figure  3  Hyperspectral  Midwave  Camera 


Hvperspectral  Camera 

Figure  3  is  a  picture  of  one  hyperspectral 
IR  camera.  This  particular  camera  uses  a 
256  x  256  HgCdTe  focal  plane  array 
integrated  with  a  cryoengine.  The  lens 
system  uses  commercial  optics  and  projects 
four  duplicate  images  on  each  of  the  four 
1 28  x  1 28  quadrants  of  the  detector  array. 


By  using  two  separate  cold  filters  inside  the 
dewar,  one  over  each  half  of  the  array, 
simultaneous  images  can  be  generated  in 
each  of  the  quadrants  of  the  Focal  Plane 
Array  (FPA)  as  illustrated  in  Figure  4.  The 
left  two  images  pass  thru  a  spectral  filter 
centered  about  the  lower  portion  (3.87m  - 
4.15  |i.)of  the  midwave  band  and  the  right 
two  images  through 


a  spectral  filter  centered  around  the  upper 
portion  of  the  midwave  band  (4.59p  -  4.72 
(i).  A  custom  lens  was  designed  for  this 
system.  The  field  of  view  of  the  lens  is  on 
the  order  of  90  degrees  square.  It  is  a 
diffraction  limited  design  and  composed  of  a 
single  aspheric  element.  The  lens  was 
procured  from  an  optics’  vendor  and  is 
composed  of  a  2x2  array  of  optical  elements 


on  a  single  optical  substrate  of  silicon.  Each 
lens  element  is  mounted  to  the  dewar  and 
centered  over  one  of  the  four  quadrants  of 
the  FPA  resulting  in  a  total  of  four  identical 
images  out  of  the  sensor.  In  order  to 
prevent  crosstalk  between  the  channels,  a 
special  cruciform  cold  shield  is  used  inside 
the  dewar  assembly  along  with  a  pair  of 
semicircular  cold  filters.  Each  cold  filter 
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covers  half  of  the  focal  plane  so  each  half  of 
the  plane  is  looking  into  one  of  the  two 
desired  wave  bands.  It  was  critical  that  the 
dewar  design  be  matched  to  the  lens  design 
for  this  system.  This  technique  of 
generating  multiple  band  IR  imagery  can  be 
expanded  to  cover  additional  bands,  and  if 
the  detector  array  is  a  dual  midwave/ 
longwave  sensor,  it  can  cover  both  the 
midwave  and  longwave  bands. 

The  camera  electronics  uses  a  modifications 
to  existing  electronics  boards.  This  primarily 
involved  altering  the  camera  electronics  to 
allow  for  two  sets  of  calibration  coefficients 
with  a  256x256  focal  plane  array.  In 
addition,  minor  modifications  are  required  to 
optimize  the  display  of  the  video  data  as  well 
as  insuring  that  the  RS422  digital  interface  is 
functioning  properly.  Finally,  interface 
cables  interfacing  the  cryoengine  and  the 
preamplifier  to  the  electronics  are  required. 

The  software  modifications  required  were  to 
the  data  collection  and  the  calibration  of  the 
two  color  data.  The  software  design 
interfaces  with  some  of  the  electronic 

changes  to  sum  multiple  frames  of  data  at 
two  different  flux  levels  in  order  to  calculate 
the  gain  and  offset  coefficients  for  each 
detector. 

The  system  integration  was  accomplished  in 
three  steps:  1)  integrated  dewar  assembly 
to  electronics,  2)  Lens  to  dewar,  and  3) 
Software  verification  with  lens  focus.  A 
custom  preamplifier  board  was  built  and 
tested  and  integrated  with  the  dewar  and 
existing  electronics  and  verified  that  the 
dewar  cooled  and  FPA  operated  as 
expected.  An  external  filter  was  also  used  to 
verify  which  band  in  the  image  was  the  red 
band  and  which  one  was  the  blue  one.  The 
lens  is  connected  to  the  optical  train  via  a 
precision  alignment  pin  insuring  repeatable 
performance  when  lens  and  dewar  are 
attached.  A  number  of  changes  were  made 
to  existing  single  color  software  to  work  with 
the  two  color  system  such  as  two  separate 
calibrations,  blacking  out  video  in  quadrants 
where  no  image  is  present  to  reduce 
distraction,  and  modifying  software  and 


electronics  to  accommodate  two  separate 
threshold  values  (one  for  each  color  band). 


Figure  5.  Camera  Output  Image 


EXAMPLE  OF  OUTPUT  IMAGERY: 

Each  256  x  256  field  out  of  the  camera 
contains  four  128  x  128  images  as  shown  in 
Figure  5.  The  two  hyperspectral  images  on 
the  top  are  time  coincident  and  the  two  on 
the  bottom  are  time  coincident  but  the  top 
and  bottom  images  differ  in  time  by  one  half 
a  frame  interval.  This  is  because  the 
multiplexer  reads  the  FPA  out  a  row  at  a 
time.  Each  256  x  256  frame  contains  two 
sets  of  hyperspectral  data  and  one  set  of 
temporal  data.  There  is  a  lot  of  processing 
that  can  be  performed  on  this  single  frame  of 
data. 

SPECTRAL  DIFFERENCE  PROCESSING: 

One  approach  to  discriminating  targets  from 
background  is  subtracted  from  the  upper  left 
image  from  upper  right  image  or  lower  left 
from  lower  right.  This  gives  a  measure  of 
change  in  energy  per  fixed  wavelength 
change  in  each  pixel.  Figure  6a  shows  the 
result  of  subtracting  the  upper  two  images 
and  figure  6b  the  result  of  subtracting  the 
upper  left  from  the  lower  right.  The 
subtraction  operation  involves  multiplying 
one  image  by  a  constant  so  both  images 
have  the  same  mean  pixel  level,  subtracting 
the  two  images  and  than  adding  1 28  so  all 
pixel  values  are  positive.  The  two  images 
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are  different  because  the  two  subframes  the  person  gave  different  residuals  pixel 

used  to  generate  Figure  6b  are  not  time  intensities  implying  they  changed  differently 

coincident  while  those  used  to  generate  across  the  sensor  waveband. 

Figure  6(a)  were  time  coincident.  It  can  be 
scene  in  Figure  6a,  that  the  background  and 


(3)  (b) 

Figure  6.  Diffeence  Images,  (1)  Upper  left  from  Upper  right,  (b)  Upper  left  from  Lower  Right 


SPECTRAL  AND  TIME  DIFFERENCING 

Spectral  differencing  of  the  images  doesn’t 
exploit  all  the  information  in  the  images  since 
the  image  includes  both  spectral  and 
temporal  information.  Figure  7  shows  the 
result  of  computing  both  the  upper  and  lower 


Figure  7.  Spectral-Temporal  Processing 

spectral  difference  images  and  than 
differencing  these  two  images  which  differ  in 
time  by  one  half  a  field  readout  time  (1/120 
sec).  This  processing  approach  highlights 
objects  that  are  moving  in  waveband  and 
time.  In  the  case  of  Figure  7,  the  persons 
lower  arms  have  a  higher  intensity  than  in 
figure  6a  and  the  background  and  body  of 
the  person  are  further  suppressed.  The 


order  of  the  temporal  and  spectral 
processing  can  be  reversed  and  for  certain 
targets,  this  might  give  an  advantage. 

Fusion  of  Multiple  Bands  at  Image  Level 

The  new  technique  of  image  level  fusion 
overcomes  one  of  the  objections  to  this 
approach  to  generating  a  hyperspectral 
camera  and  that  is  the  loss  in  spatial 
resolution  since  the  array  has  256x256 
pixels  but  each  subpicture  has  only  128x128 
pixels.  The  image  resolution  has  been 
halved  in  both  the  horizontal  and  vertical 
directions.  The  direct  approach  to 
recovering  the  original  resolution  requires 
going  to  larger  arrays  or  narrower  field-of- 
view  optics. 

Figure  8  shows  what  happens  when  the  four 
images  are  combined  in  a  simple  manner  by 
reordering  the  pixels  so  similar  pixels  are 
placed  next  to  each  other  in  a  composite 
256x256  image.  The  256x256  looks  blurry 
because  there  is  some  misregistration 
between  the  four  subframes.  If  adjacent 
pixels  are  combined  using  a  weighting  filter 
were  the  weighting  factor  is  the  primary 
colors,  the  blurring  will  be  removed  and  the 
image  will  become  a  color  image  where 
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Figure  8.  Fused  Hyperspectral  Image 

color  relates  to  color  illustrated  in  Figure  9. 

If  the  four  images  had  been  offset  from  each 
other  by  a  half  pixel  shift  in  different 
combinations  of  up,  down,  right  and  left,  the 
resulting  image  would  look  as  if  it  were 
created  by  scanning  the  array.  This 
suggests  that  techniques  developed  for 
deblurring  images  could  be  used  for 
recovering  the  original  256x256  spatial 
resolution. 

Figure  10  illustrates  one  way  the  four  images 
can  be  combined  in  support  of  resolution 


enhancement  processing  in  they  were  offset 
by  half  a  pixel.  Figure  1 0  uses  the  upper  left 
image  as  a  reference  for  a  starting  place. 
The  upper  right  image  is  shifted  from  the 
upper  left  image  by  a  half  a  pixel  right.  The 
lower  left  image  is  shifted  down  by  half  a 
pixel  and  the  lower  right  image  both  a  half 
pixel  down  and  to  the  right.  Pixels  in  the  four 
images  are  taken  to  form  new  composite 
pixels  as  shown  in  Figure  10.  This 
composite  image  is  a  blurred  form  of  any  of 
the  four  original  images.  Any  of  several 
deblurring  algorithms  can  be  used  to  obtain 
the  desired  resolution  improvement.  The 


Figure  10  Image  Fusion  With  Resolution  Enhancement 
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pixel  shift  so  the  processing  cannot  be  done 
using  the  existing  imagery.  A  new  camera 
couid  provide  the  overlapping  imagery. 

CONCLUSIONS 

A  new  technique  for  image  level  fusion  of 
hyperspectral  processing  has  been 
presented  which  makes  additional  use  of  the 
hyperspectral  imagery  if  the  images  have  the 
required  shift  with  respect  to  each  other. 

The  technique  can  be  expanded  to  cover  as 
many  bands  as  the  sensor  can  provide. 
Additional  bands  (images)  benefits  the 
resolution  enhancement  processing  if  the 
image  to  image  shift  is  adjust  to  provide 
additional  spatial  sampling  of  the  sensor 
spot  function. 
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